from sklearn_benchmarks.reporting.hp_match import HPMatchReporting

scikit-learn vs. ONNX Runtime

reporting = HPMatchReporting(against_lib="onnx", config="config.yml")
reporting.make_report()

We assume here there is a perfect match between the hyperparameters of both librairies. For a given set of parameters and a given dataset, we compute the speedup time scikit-learn / time onnx. For instance, a speedup of 2 means that onnx is twice as fast as scikit-learn for a given set of parameters and a given dataset.

KNeighborsClassifier_brute_force

onnx (1.10.1) vs. scikit-learn (1.0.dev0)

Speedup barplots

All estimators share the following parameters: algorithm=brute.

WARNING! Mismatch between validation scores for 12 predictions (75.0%). See below.

	estimator	function	n_samples_train	n_samples	n_features	accuracy_score_sklearn	accuracy_score_onnx	diff_accuracy_scores
1	KNeighborsClassifier_brute_force	predict	100000	1000	100	0.663	1.000	0.337
2	KNeighborsClassifier_brute_force	predict	100000	1	100	1.000	0.757	0.243
4	KNeighborsClassifier_brute_force	predict	100000	1000	100	0.757	0.882	0.125
7	KNeighborsClassifier_brute_force	predict	100000	1000	100	0.882	1.000	0.118
8	KNeighborsClassifier_brute_force	predict	100000	1	100	1.000	0.757	0.243
10	KNeighborsClassifier_brute_force	predict	100000	1000	100	0.882	0.663	0.219
13	KNeighborsClassifier_brute_force	predict	100000	1000	100	0.757	1.000	0.243
14	KNeighborsClassifier_brute_force	predict	100000	1	100	1.000	0.922	0.078
16	KNeighborsClassifier_brute_force	predict	100000	1000	100	0.663	0.929	0.266
19	KNeighborsClassifier_brute_force	predict	100000	1000	2	0.896	1.000	0.104
20	KNeighborsClassifier_brute_force	predict	100000	1	2	1.000	0.922	0.078
22	KNeighborsClassifier_brute_force	predict	100000	1000	2	0.922	0.896	0.026

Raw results

	estimator	function	n_samples_train	n_samples	n_features	mean_duration_sklearn	std_duration_sklearn	iteration_throughput	latency	n_jobs	n_neighbors	mean_duration_onnx	std_duration_onnx	accuracy_score_onnx	speedup	std_speedup	sklearn_profiling	onnx_profiling
0	KNeighborsClassifier_brute_force	fit	100000	100000	100	0.016	0.001	4.883	0.0	-1	1	21.402	0.125	0.663	0.001	0.001	See	See
3	KNeighborsClassifier_brute_force	fit	100000	100000	100	0.016	0.000	4.937	0.0	-1	5	0.389	0.011	1.000	0.042	0.042	See	See
6	KNeighborsClassifier_brute_force	fit	100000	100000	100	0.016	0.001	4.878	0.0	1	100	21.940	0.090	0.882	0.001	0.001	See	See
9	KNeighborsClassifier_brute_force	fit	100000	100000	100	0.016	0.000	5.032	0.0	-1	100	0.390	0.014	1.000	0.041	0.041	See	See
12	KNeighborsClassifier_brute_force	fit	100000	100000	100	0.017	0.001	4.810	0.0	1	5	6.605	0.110	0.896	0.003	0.003	See	See
15	KNeighborsClassifier_brute_force	fit	100000	100000	100	0.017	0.001	4.846	0.0	1	1	0.284	0.012	1.000	0.058	0.058	See	See
18	KNeighborsClassifier_brute_force	fit	100000	100000	2	0.005	0.000	0.345	0.0	-1	1	6.589	0.136	0.929	0.001	0.001	See	See
21	KNeighborsClassifier_brute_force	fit	100000	100000	2	0.005	0.000	0.324	0.0	-1	5	0.298	0.013	1.000	0.017	0.017	See	See

	estimator	function	n_samples_train	n_samples	n_features	mean_duration_sklearn	std_duration_sklearn	iteration_throughput	latency	n_jobs	n_neighbors	mean_duration_onnx	std_duration_onnx	accuracy_score_onnx	speedup	std_speedup	sklearn_profiling	onnx_profiling
1	KNeighborsClassifier_brute_force	predict	100000	1000	100	2.571	0.223	0.0	0.003	-1	1	0.401	0.011	1.000	6.414	6.417	See	See
2	KNeighborsClassifier_brute_force	predict	100000	1	100	0.036	0.005	0.0	0.036	-1	1	21.310	0.233	0.757	0.002	0.002	See	See
4	KNeighborsClassifier_brute_force	predict	100000	1000	100	3.395	0.047	0.0	0.003	-1	5	21.733	0.037	0.882	0.156	0.156	See	See
5	KNeighborsClassifier_brute_force	predict	100000	1	100	0.035	0.003	0.0	0.035	-1	5	0.411	0.010	1.000	0.086	0.086	See	See
7	KNeighborsClassifier_brute_force	predict	100000	1000	100	2.683	0.028	0.0	0.003	1	100	0.394	0.013	1.000	6.805	6.808	See	See
8	KNeighborsClassifier_brute_force	predict	100000	1	100	0.031	0.002	0.0	0.031	1	100	21.015	0.076	0.757	0.001	0.001	See	See
10	KNeighborsClassifier_brute_force	predict	100000	1000	100	3.457	0.049	0.0	0.003	-1	100	21.038	0.422	0.663	0.164	0.164	See	See
11	KNeighborsClassifier_brute_force	predict	100000	1	100	0.036	0.005	0.0	0.036	-1	100	0.388	0.014	1.000	0.092	0.092	See	See
13	KNeighborsClassifier_brute_force	predict	100000	1000	100	2.688	0.028	0.0	0.003	1	5	0.309	0.007	1.000	8.687	8.689	See	See
14	KNeighborsClassifier_brute_force	predict	100000	1	100	0.033	0.003	0.0	0.033	1	5	6.785	0.016	0.922	0.005	0.005	See	See
16	KNeighborsClassifier_brute_force	predict	100000	1000	100	1.822	0.012	0.0	0.002	1	1	6.658	0.119	0.929	0.274	0.274	See	See
17	KNeighborsClassifier_brute_force	predict	100000	1	100	0.032	0.003	0.0	0.032	1	1	0.298	0.010	1.000	0.106	0.106	See	See
19	KNeighborsClassifier_brute_force	predict	100000	1000	2	2.097	0.068	0.0	0.002	-1	1	0.290	0.006	1.000	7.239	7.240	See	See
20	KNeighborsClassifier_brute_force	predict	100000	1	2	0.006	0.004	0.0	0.006	-1	1	6.569	0.117	0.922	0.001	0.001	See	See
22	KNeighborsClassifier_brute_force	predict	100000	1000	2	3.053	0.063	0.0	0.003	-1	5	6.664	0.090	0.896	0.458	0.458	See	See
23	KNeighborsClassifier_brute_force	predict	100000	1	2	0.006	0.001	0.0	0.006	-1	5	0.307	0.009	1.000	0.021	0.021	See	See

KNeighborsClassifier_kd_tree

onnx (1.10.1) vs. scikit-learn (1.0.dev0)

Speedup barplots

All estimators share the following parameters: algorithm=kd_tree.

WARNING! Mismatch between validation scores for 12 predictions (75.0%). See below.

	estimator	function	n_samples_train	n_samples	n_features	accuracy_score_sklearn	accuracy_score_onnx	diff_accuracy_scores
1	KNeighborsClassifier_kd_tree	predict	1000000	1000	10	0.929	1.000	0.071
2	KNeighborsClassifier_kd_tree	predict	1000000	1	10	1.000	0.946	0.054
4	KNeighborsClassifier_kd_tree	predict	1000000	1000	10	0.946	0.951	0.005
7	KNeighborsClassifier_kd_tree	predict	1000000	1000	10	0.951	1.000	0.049
8	KNeighborsClassifier_kd_tree	predict	1000000	1	10	1.000	0.946	0.054
10	KNeighborsClassifier_kd_tree	predict	1000000	1000	10	0.951	0.929	0.022
13	KNeighborsClassifier_kd_tree	predict	1000000	1000	10	0.946	1.000	0.054
14	KNeighborsClassifier_kd_tree	predict	1000000	1	10	1.000	0.911	0.089
16	KNeighborsClassifier_kd_tree	predict	1000000	1000	10	0.929	0.894	0.035
19	KNeighborsClassifier_kd_tree	predict	1000	1000	2	0.891	1.000	0.109
20	KNeighborsClassifier_kd_tree	predict	1000	1	2	1.000	0.911	0.089
22	KNeighborsClassifier_kd_tree	predict	1000	1000	2	0.911	0.891	0.020

Raw results

	estimator	function	n_samples_train	n_samples	n_features	mean_duration_sklearn	std_duration_sklearn	iteration_throughput	latency	n_jobs	n_neighbors	mean_duration_onnx	std_duration_onnx	accuracy_score_onnx	speedup	std_speedup	sklearn_profiling	onnx_profiling
0	KNeighborsClassifier_kd_tree	fit	1000000	1000000	10	3.358	0.105	0.024	0.0	-1	1	165.148	0.000	0.929	0.020	0.020	See	See
3	KNeighborsClassifier_kd_tree	fit	1000000	1000000	10	4.787	0.058	0.017	0.0	-1	5	3.236	0.276	1.000	1.479	1.484	See	See
6	KNeighborsClassifier_kd_tree	fit	1000000	1000000	10	4.429	0.102	0.018	0.0	1	100	163.326	0.000	0.951	0.027	0.027	See	See
9	KNeighborsClassifier_kd_tree	fit	1000000	1000000	10	4.361	0.084	0.018	0.0	-1	100	3.362	0.297	1.000	1.297	1.302	See	See
12	KNeighborsClassifier_kd_tree	fit	1000000	1000000	10	4.405	0.187	0.018	0.0	1	5	0.064	0.004	0.891	69.053	69.166	See	See
15	KNeighborsClassifier_kd_tree	fit	1000000	1000000	10	4.292	0.110	0.019	0.0	1	1	0.006	0.000	1.000	751.801	753.797	See	See
18	KNeighborsClassifier_kd_tree	fit	1000	1000	2	0.001	0.001	0.015	0.0	-1	1	0.090	0.005	0.894	0.012	0.012	See	See
21	KNeighborsClassifier_kd_tree	fit	1000	1000	2	0.001	0.000	0.023	0.0	-1	5	0.006	0.000	1.000	0.113	0.114	See	See

	estimator	function	n_samples_train	n_samples	n_features	mean_duration_sklearn	std_duration_sklearn	iteration_throughput	latency	n_jobs	n_neighbors	mean_duration_onnx	std_duration_onnx	accuracy_score_onnx	speedup	std_speedup	sklearn_profiling	onnx_profiling
1	KNeighborsClassifier_kd_tree	predict	1000000	1000	10	1.022	1.203	0.0	0.001	-1	1	3.325	0.239	1.000	0.308	0.308	See	See
2	KNeighborsClassifier_kd_tree	predict	1000000	1	10	0.004	0.001	0.0	0.004	-1	1	161.935	0.000	0.946	0.000	0.000	See	See
4	KNeighborsClassifier_kd_tree	predict	1000000	1000	10	1.345	0.448	0.0	0.001	-1	5	164.204	0.000	0.951	0.008	0.008	See	See
5	KNeighborsClassifier_kd_tree	predict	1000000	1	10	0.004	0.001	0.0	0.004	-1	5	3.293	0.251	1.000	0.001	0.001	See	See
7	KNeighborsClassifier_kd_tree	predict	1000000	1000	10	7.718	0.702	0.0	0.008	1	100	3.254	0.309	1.000	2.372	2.382	See	See
8	KNeighborsClassifier_kd_tree	predict	1000000	1	10	0.004	0.001	0.0	0.004	1	100	161.889	0.000	0.946	0.000	0.000	See	See
10	KNeighborsClassifier_kd_tree	predict	1000000	1000	10	3.956	0.413	0.0	0.004	-1	100	165.548	0.000	0.929	0.024	0.024	See	See
11	KNeighborsClassifier_kd_tree	predict	1000000	1	10	0.006	0.001	0.0	0.006	-1	100	3.414	0.276	1.000	0.002	0.002	See	See
13	KNeighborsClassifier_kd_tree	predict	1000000	1000	10	2.260	0.540	0.0	0.002	1	5	0.006	0.000	1.000	357.440	358.236	See	See
14	KNeighborsClassifier_kd_tree	predict	1000000	1	10	0.002	0.001	0.0	0.002	1	5	0.070	0.007	0.911	0.025	0.025	See	See
16	KNeighborsClassifier_kd_tree	predict	1000000	1000	10	1.286	0.378	0.0	0.001	1	1	0.086	0.005	0.894	14.924	14.950	See	See
17	KNeighborsClassifier_kd_tree	predict	1000000	1	10	0.001	0.000	0.0	0.001	1	1	0.006	0.001	1.000	0.199	0.199	See	See
19	KNeighborsClassifier_kd_tree	predict	1000	1000	2	0.035	0.019	0.0	0.000	-1	1	0.006	0.000	1.000	5.543	5.554	See	See
20	KNeighborsClassifier_kd_tree	predict	1000	1	2	0.003	0.001	0.0	0.003	-1	1	0.064	0.003	0.911	0.044	0.044	See	See
22	KNeighborsClassifier_kd_tree	predict	1000	1000	2	0.034	0.003	0.0	0.000	-1	5	0.065	0.003	0.891	0.534	0.535	See	See
23	KNeighborsClassifier_kd_tree	predict	1000	1	2	0.003	0.000	0.0	0.003	-1	5	0.006	0.000	1.000	0.465	0.466	See	See

HistGradientBoostingClassifier_best

onnx (1.10.1) vs. scikit-learn (1.0.dev0)

Speedup barplots

All estimators share the following parameters: learning_rate=0.01, n_iter_no_change=10.0, max_leaf_nodes=100.0, max_bins=255.0, min_samples_leaf=100.0, max_iter=300.0.

WARNING! Mismatch between validation scores for 1 prediction (100.0%). See below.

	estimator	function	n_samples_train	n_samples	n_features	accuracy_score_sklearn	accuracy_score_onnx	diff_accuracy_scores
1	HistGradientBoostingClassifier_best	predict	100000	1000	100	0.824	1.0	0.176

Raw results

	estimator	function	n_samples_train	n_samples	n_features	mean_duration_sklearn	std_duration_sklearn	n_iter	iteration_throughput	latency	mean_duration_onnx	std_duration_onnx	accuracy_score_onnx	speedup	std_speedup	sklearn_profiling	onnx_profiling
0	HistGradientBoostingClassifier_best	fit	100000	100000	100	131.475	0.0	300	0.001	0.001	0.598	0.03	0.824	219.824	220.104	See	See

	estimator	function	n_samples_train	n_samples	n_features	mean_duration_sklearn	std_duration_sklearn	n_iter	iteration_throughput	latency	mean_duration_onnx	std_duration_onnx	accuracy_score_onnx	speedup	std_speedup	sklearn_profiling	onnx_profiling
1	HistGradientBoostingClassifier_best	predict	100000	1000	100	0.187	0.004	300	0.004	0.0	0.522	0.019	1.0	0.358	0.358	See	See